1. Field of the Invention
The present invention relates to an image sensing apparatus equipped with an image sensor having a photoelectric conversion characteristic comprising a plurality of different characteristic regions, such as a photoelectric conversion characteristic which comprises a linear characteristic region where an electric signal is output therefrom while being transformed linearly relative to an incident light quantity and a logarithmic characteristic region where an electric signal is output therefrom while being transformed logarithmically relative to the incident light quantity, more specifically, a photoelectric conversion characteristic where a rate of change relative to the incident light quantity in a high luminance region is less than that in a low luminance region. In particular, the present invention relates to an image processing method for use in processing an image picked up by the image sensor, and an image sensing apparatus employing the image processing method.
2. Description of the Related Art
Late years, in order to cope with the demand for higher image quality, the expansion in a luminance range (brightness range) of a subject, or a dynamic range, which is treatable or processable by an image sensor, has become one of major themes in image sensing apparatuses, such as digital cameras. In connection with the expansion of a dynamic range, there has been known an image sensor designed to add a logarithmic transformation circuit having a MOSFET etc., to a solid-state image sensing device having a plurality of photoelectric conversion elements, such as photodiodes, disposed in a matrix arrangement, so as to allow the solid-state image sensing device to have an output characteristic which comprises one region where an electric signal is output therefrom while being transformed linearly relative to an incident light quantity and the other region where an electric signal is output therefrom while being transformed logarithmically relative to the incident light quantity, by taking advantage of a sub-threshold characteristic of the MOSFET (This image sensor is known and described as “LN/LOG sensor”. An image having linear and logarithmic characteristic regions obtained by the LN/LOG sensor will hereinafter be referred to as “linear/logarithmic image”). As mentioned above, this LN/LOG sensor can provide an output transformed natural-logarithmically relative to an incident light quantity. Thus, as compared with an image sensor having a photoelectric conversion characteristic consisting only of a linear characteristic region, the LN/LOG sensor makes it possible to ensure a wider dynamic range. While this capability of ensuring a wider dynamic range is advantageous, an image signal is logarithmically compressed to cause a problem about deterioration in contrast (tone or gradation), particularly, in a logarithmic characteristic region.
Techniques, such as the LN/LOG sensor, for providing a wider dynamic range in an image sensing system are being developed. In contrast to the image sensing system, the development for providing a wider dynamic range in a display system (e.g. display or monitor) has less progressed so far. Thus, even if a dynamic range is expanded in an image to be input to the display system, this advantage is likely to be unable to be reflected in display. Further, an image having a wider dynamic range (wider dynamic range image) allows a larger volume of information than ever before to be contained therein. In this case, if such an image is handled (transmitted or stored) in a conventional manner, an image sensing apparatus will inevitably suffer from increase in cost. In order to effectively utilize the image data including a larger volume of information, or to allow an image obtained from an image sensing system using advanced techniques of expanding a dynamic range to be displayed with high image quality by use of a display system using less advanced techniques of expanding a dynamic range, it is requited to compress the entire dynamic range while maintaining valuable information contained in a wider dynamic range image.
In the above background, for example, the following Non-Patent Publication 1 discloses a dynamic range compression (hereinafter referred to as “DR compression) technique (multi-scale Retinex) based on the Retinex theory with a focus on human ocular characteristics (color constancy to luminance or brightness, which is intended to extract from an input image a component arising from an intensity of illumination light (this component will hereinafter be referred to as “illumination component”), and compress the illumination component so as to compress the entire dynamic range while maintaining a local contrast.
As to the color constancy, while light reflected from any of various objects is logically derived as a product of a reflectance of an object surface and illumination light illuminating the object surface, an actually perceived color and brightness of the object is largely dependent on the surface reflectance rather than characteristic of light projected on the retina. That is, the human ocular system has a mechanism of subtracting or eliminating an influence of the illumination light from the image on the retina. For example, respective spectral changes of an object under natural sunlight and under incandescent bulb light are perceived as a change in the illumination light but not as a change in color of the object itself. Based on this concept of color constancy, the Retinex theory is intended to extract an illumination component from an input image, and attenuate the extracted illumination component while maximally maintain a component arising from a surface reflectance to be noticed by the ocular system (this component will hereinafter be referred to as “reflectance component”), so as to reproduce a high-contract image. [Non-Patent Publication 1: “Image Dynamic Range Compression Techniques”, IMAGE LAB (June 2004) pp 24 to 28.
If the DR compression technique as disclosed in the Non-Patent Publication 1 is applied to a linear/logarithmic image, and a wide dynamic range image obtained by a LN/LOG sensor is subjected to DR compression for providing improved (enhanced) contrast therein. This has a potential for allowing the linear/logarithmic image to have improved image quality (higher image quality) in a display system. However, the DR compression technique is designed for an image obtained through photoelectric conversion in a linear characteristic (linear characteristic image), and any compression technique for a linear/logarithmic image is not disclosed in the Non-Patent Publication 1.
When the DR compression technique is applied to a linear/logarithmic image, it is necessary to take into account, for example, the following points (1) and (2).
(1): If the DR compression technique is applied to a linear/logarithmic image without modifications, the linear/logarithmic image is simply compressed over all in such a manner that a photoelectric conversion characteristic 901 is transformed into a photoelectric conversion characteristic 902, as shown, for example, in FIG. 23. Thus, even though contrast in a logarithmic characteristic region is slightly improved (contrast in a linear characteristic region is further highlighted), each contrast in the logarithmic and linear characteristic regions has almost no change in terms of the entire photoelectric conversion characteristic (as compared with a photoelectric conversion characteristic 903 transformed without using the DR compression), and a desired contrast improvement is not achieved.
(2): As shown in FIG. 24A, when a logarithmic characteristic (image) in a linear/logarithmic image is transformed into a linear characteristic and the image is subjected to DR compression, an integer operation is performed during the transformation into the linear characteristic (a fractional part of a number is truncated, for example, “0.2” of “40.2” is truncated to be “40”), and consequently contrast in the linear characteristic 911 after the transformation will be lost. Moreover, as shown in FIG. 24B, if a logarithmic characteristic is transformed into linear characteristic while avoiding the above deterioration in contrast of the transformed linear characteristic, it is required to increase a bit width in an image processing section, as compared with a processing for the linear/logarithmic image illustrated in 24A. This leads to increase in device cost.